fix: reset frame when read to EOF to release buffer in Block #220

aiquestion · 2024-08-19T11:48:45Z

We use reader to decompress with no pooling object, it's some code like:

func Decompress(compressed []byte) ([]byte, error) {
	if len(compressed) == 0 {
		return compressed, nil
	}
	destBuffer := bytes.Buffer{}
	bytesReader := bytes.NewReader(compressed)
	lz4Reader := lz4.NewReader(bytesReader)
	if _, err := destBuffer.ReadFrom(lz4Reader); err != nil {
		return nil, err
	}
	return destBuffer.Bytes(), nil
}

and we found a big performace regression when we upgrade to v4.1.21 (from v4.0.2), after some test we narrow down the change from v4.1.0 -> v4.1.1.

It's seems that the code do BlockSizeIndex.Get() twice and only call Put() to return the buffer 1 time. It look like that the buffer from this stack is not released.

lz4block.BlockSizeIndex.Get (blocks.go:54) github.com/pierrec/lz4/v4/internal/lz4block
lz4stream.NewFrameDataBlock (block.go:197) github.com/pierrec/lz4/v4/internal/lz4stream
lz4stream.(*Blocks).initR (block.go:99) github.com/pierrec/lz4/v4/internal/lz4stream
lz4stream.(*Frame).InitR (frame.go:114) github.com/pierrec/lz4/v4/internal/lz4stream
lz4.(*Reader).init (reader.go:91) github.com/pierrec/lz4/v4
lz4.(*Reader).Read (reader.go:111) github.com/pierrec/lz4/v4
bytes.(*Buffer).ReadFrom (buffer.go:211) bytes

So here i added a Reset when read to EOF, to release the underlying buffer in Block.
Added a Benchmark test:

Before:
BenchmarkReaderNoReset-12    	  258943	      4354 ns/op	   70309 B/op	      12 allocs/op

After:
BenchmarkReaderNoReset-12    	 1136524	      1031 ns/op	    4226 B/op	      11 allocs/op

fix: reset frame when read to EOF to release buffer in Block

72f5eb6

aiquestion mentioned this pull request Aug 19, 2024

Did not return byte buffer when reader read to EOF ( from v4.1.1) #221

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: reset frame when read to EOF to release buffer in Block #220

fix: reset frame when read to EOF to release buffer in Block #220

aiquestion commented Aug 19, 2024 •

edited

Loading

fix: reset frame when read to EOF to release buffer in Block #220

Are you sure you want to change the base?

fix: reset frame when read to EOF to release buffer in Block #220

Conversation

aiquestion commented Aug 19, 2024 • edited Loading

aiquestion commented Aug 19, 2024 •

edited

Loading